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Abstract 

Course Management Systems are becoming increasingly accepted as an effective and efficient 
method of course delivery. Often they are used as the principal means of web -based course de- 
livery. In addition, many institutions are utilizing CMSs to more economically handle large sec- 
tions of particular courses. However, CMSs can also be used as a valuable tool for supplement- 
ing face-to-face course instruction. Although we have used a CMS extensively in large sections 
of lower-level courses, we have not, until now, made a concentrated effort at examining the ef- 
fect of using a CMS in our upper-level computer science programming courses. This paper will 
focus on the pedagogical uses of a CMS and the specific aspects of a CMS that seem to either 
contribute to or detract from student learning. We also explore the features we would like to em- 
ploy in the future that may increase student motivation and success for our computer science stu- 
dents. 

Introduction 

As educators, we are (or at least should be) focused on the process of student learning. We 
should be developing innovative ways to teach students how to learn, not just using technology 
as an alternate method of supplying the same information to students. With this in mind, we have 
begun to examine the ways in which a course management system can be utilized, and perhaps 
exploited, to complement face-to-face instruction, with the ultimate goal being an improvement 
in student learning through enhanced problem-solving skills. One of the components of the 
course management system that we are especially interested in, because we believe it is an essen- 
tial part of the learning process, is the set of tools available to promote communication among 
students, as well as with the instructor. This feature, if utilized effectively, can promote collabo- 
rative learning, and thus, enhance critical thinking s ki lls [4]. 

Background 

A number of studies, including our own, have been performed to determine the effect of exten- 
sive use of technology enhancements in higher education settings [3, 5, 6, 7, 9, 10]. The studies 
vary from trying to determine if CMSs can be used to reduce costs to more effectively dealing 
with large lecture courses. A study at UC Berkeley was conducted to determine the effect of 
technology enhancements in large sections of Chemistry 1A, with the goal being to determine if 
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the use of a CMS affected the course cost [6]. Faculty members at a number of institutions are 
viewing CMSs as a means to deal with large lecture classes more effectively and determine if the 
use of a CMS might impact student motivation and success [3, 7]. And finally, more and more 
computer science departments are experimenting with CMSs in the large introductory computer 
science courses [9]. 

Our research began in two arenas. The first was that of a large lecture-based computer science 
course taught primarily for business majors. The second was that of a beginning programming 
course. The primary purpose of our prior research was to determine if and how student success is 
affected by a CMS [5]. 

We have since extended our research to include an examination of utilizing a CMS in our upper- 
level computer science programming courses. We wanted to determine if there were specific 
pedagogical methodologies that could be employed around the use of a CMS which might either 
contribute to or detract from student learning. If the use of a CMS proves to be successful, we 
want to explore the specific features that did in fact seem to increase student motivation, prob- 
lem-solving skills, and success of our computer science students. 

Research Methods 

The study was conducted using three classes. The first course, CSC 121, is an introductory 
course which is required for all business majors. The typical student profile is primarily fresh- 
man, pre-business majors with little or no computer experience. This course has ut ili zed WebCT 
for several years with some positive results [5]. The second course, CSC 241, is the Data Struc- 
tures course which is taken by all Computer Science and Computer Information Systems majors. 
It is typically the 3 rd , 4 th ' or 5 th computer science course taken in the curriculum. In addition to 
using the system hosting WebCT, the students develop programs and submit them on a Linux 
system. The third course, CSC 321, is the Programming Methods for Business Applications 
course, which has, in the past, been taught using COBOL. As a result of recent curriculum revi- 
sions, the course was taught this semester using Visual Basic .NET, so the students were working 
in an entirely new environment. Despite the fact that the language itself is new to the students, 
they are junior-level computer science majors, therefore were expected to be well-versed in prob- 
lem-solving methods and programming language constructs. 

As a result of previous studies, we noted that it would be useful to administer some type of pre- 
course/end-of-course survey [5]. The purpose of the pre-course survey is to give some indication 
as to the level of computer literacy each student possesses, some idea of their work habits in 
terms of school work, and a measure of their familiarity and attitude regarding online and tech- 
nology-enhanced courses. We also conducted an end-of-course survey to determine whether the 
use of WebCT in the curriculum had changed student attitudes and responses to any of the fac- 
tors. 
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The Survey 

The pre-course survey was administered during the first week of classes. Students were asked to 
participate in an anonymous online survey using WebCT. The WebCT survey tool was utilized 
because the instructor could determine who responded to the survey but could not attribute re- 
sponses to a specific student. The students were advised of the way a survey in WebCT works 
and were informed that the survey counted as a quiz - full credit if they took the survey, no 
credit if they did not take the survey. Of the 79 students enrolled in CSC 121, 74 responded to 
the survey. The CSC 241 class had 25 students enrolled, 24 of whom responded, and the CSC 
321 class, consisting of 23 students, had 100% participation in the survey. The total participation 
rate for the pre-course survey was over 95%. This in itself is quite impressive, given the fact that 
a number of these students had never before used a course management system. 

Questions on the surveys were designed and may be categorized according to the type of infor- 
mation we were trying gain. We wanted to get an idea of the computer background of the stu- 
dents, so the questions included “How would you rate your computer skills?”, “Have you ever 
used WebCT before?”, and “Have you ever used other online course enhancements or course 
management systems?”. 

From past experience, we know that many students have previously used some type of course 
management system so it is becoming quite common for them to anticipate instructor use of 
WebCT. In order to gain insight into this, we asked questions such as “Did you look at the 
WebCT site for this course prior to the first class meeting?” and “If you did access the site for 
this course prior to the first day of class, what pages did you access?”. 

In an attempt to determine whether the use of a course management system might actually in- 
crease student motivation and learning, we posed several questions regarding the features of a 
CMS. This was done in an attempt to determine if and how specific features might be exploited. 
Questions included “Given your choice, when would you work on assignments?”, “Having 
online access to assignments would [help or make little difference in my work habits]”, and 
“How much would you use chat room, discussion forum, and e-mail features if they were avail- 
able to you?” We also posed a question to determine the preferred work environment (work, 
home, or either) of the students. 

Results of Pre-Course Survey 

There is such a strong trend in business toward technology today that, more and more, students 
graduating from college will be hard-pressed to find a job that does not involve technology in 
some form or fashion. To this end, the use of information technology in education is growing 
dramatically [1, 11]. One of the survey questions was directed at measuring the students’ level of 
computer literacy. Of course, the measure was the student’s perceived level of computer literacy, 
so the results were somewhat interesting. Figure 1 represents the results of the survey question 
regarding computer literacy. 

It is interesting to note that the CSC 241 students actually rank their computer skills higher (75% 
expert or power user) than those of the CSC 321 students (56.5% expert or power user). The stu- 
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dents who ranked themselves as novice or below were all enrolled in the CSC 121 course (9.5% 
of the class). 



Figure 1. Computer Skill Rating By Class 




□ 121 

□ 241 

□ 321 



Expert Power User Average Novice Basic or 

None 



Based on the increased use of technology in the classroom, we wanted to quantify the degree of 
use of course management systems and/or some form of online course enhancements. Figure 2 
represents an overview of the level of online use students have experienced in the classroom. The 
students were asked to select any and all of the course enhancements that they had utilized in 
previous courses. 

It is apparent from the results that students at all levels are accustomed to some form of online 
submission process. Over 55% of the students have accessed a course website and a high per- 
centage of students have used WebCT prior to the current course. A smaller percentage of stu- 
dents have utilized discussion forums, chat groups, and e-mail specific to a particular course. 

Given those results, it is interesting to note the number of students actually accessing the course 
material via WebCT prior to the start of the semester. The students were asked to indicate which 
pages, if any, they visited prior to the first day of class. Approximately 30% of the students over- 
all visited the course site prior to the first day of class. Of the CSC 121 students, 37.5% visited 
the site; of the CSC 241 students, 20.8% visited the site; and of the CSC 321 students, 34.8% vis- 
ited the site. 
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Figure 2. Online Course Enhancements/CMS Used 
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Of those students who visited the course site prior to the first day of classes, the majority of them 
visited the course syllabus and calendar (see Figure 3). Almost 40% of the CSC 321 students 
who accessed the course site also accessed the assignments page. 



Figure 3. Course Pages Accessed Prior to First Day 




Calendar Syllabus Discussion Email Policies Assignments Lecture 

Chat Notes 



The final set of questions was designed to give some indication as to how the features of a CMS 
might be utilized more effectively in order to enhance student success in the course. Not surpris- 
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ingly, a large percentage of students (CSC 121 - 79.1%, CSC 241 - 79.2%, CSC 321 - 73.9%) 
indicated that they felt having online access to assignments would allow them to keep up with 
due dates, budget time, and be a more effective student. In general (and again, not surprisingly), 
the majority of students work on assignments between the hours of 3:00 p.m. and 5:00 a.m. Fig- 
ure 4 shows the distribution of times when students prefer to work on assignments. 

These statistics, for obvious reasons, have an effect on the student responses in terms of where 
they prefer to work on assignments. Over 50% of the students in the CSC 121 and CSC 241 
classes prefer to work on assignments at home, with approximately 37% in each class stating that 
they could work either in the lab or at home. Of the CSC 321 students, 30% (the highest percent- 
age in all classes) preferred to work in the lab. This may be attributed to the fact that the more 
advanced computer science students have learned the benefits of collaborative learning. The re- 
maining students either preferred to work at home or could work either in the lab or at home. 



Figure 4. When Students Prefer to Work on Assignments 




5am-10am 10am-3pm 3pm-8pm 8pm-12am 12am-5am 



We were extremely interested in the students’ opinion as to how much they would use chat 
room, discussion forum, and e-mail features if they were available. The results, shown in Figure 
5, indicate that the majority of the CSC 241 students (58%) actually prefer online tools versus 
face-to-face or telephone communication. An average number of students (CSC 121 - 45.9%, 
CSC 121 - 37.5%, CSC 321 - 52.2%) thought they would probably use them but would prefer to 
talk to a person. The remaining students stated that they would not use the tools unless required 
for the class. 

How WebCT is Utilized in Various Courses 

In all of the courses, WebCT was used for distribution of course syllabus and policies, calendar 
postings, posting of assignments, student submission of assignments, online quizzes (to a limited 
extent), posting of grades, discussion groups, chat, and e-mail. The CSC 121 students utilized 
both student web pages and student presentations for a web page development exercise. 
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Figure 5. Student Opinion of Online Communications 
Compared to Face-To-Face Communications 




The use of WebCT also varied to some degree in the CSC 241 and CSC 321 courses which are 
taken by the Computer Science and Computer Information Systems majors. One of the differ- 
ences involved the posting of lecture material. Both course sites contained pages of course in- 
formation material in the form of lecture presentations. The students were able to access the 
notes prior to class in order to study and better prepare for the class. 

Another difference in the course sites is a result of the fact that program development for the 
Data Structures course was on a separate Linux system. In a prior study [5], students in the pro- 
gramming course were sometimes confused by the programming assignments being posted cn 
the WebCT assignments page. This page indicates whether or not a particular assignment has 
been submitted. Since students were submitting programs on a different system, the WebCT as- 
signment always displayed a ‘Not Submitted’ message. To avoid confusion, the programming 
assignments are now placed on a separate content module page. Therefore the students can ob- 
tain the assignment online, but more easily recognize the fact that some assignments are submit- 
ted via WebCT, while others are submitted on the Linux machine. 

Results of End-of- Course Survey 

The post-course survey was administered a month prior to the end of classes. Again, students 
were asked to participate in an anonymous online survey using WebCT. The students were in- 
formed that the survey counted as a bonus quiz - extra quiz credit if they took the survey, no 
penalty if they did not take the survey. Of the 75 students remaining enrolled in CSC 121, 58 re- 
sponded to the survey. The CSC 241 class had 17 students remaining enrolled, 13 of whom re- 
sponded, and the CSC 321 class, consisting of 23 students, had 22 students participate in the sur- 
vey. The total participation rate for the end-of-course survey was almost 89%. Not as impressive 
as the pre-course survey participation but still quite good in terms of voluntary participation. 
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The questions on the survey were designed to mimic most of the pre-course survey questions. 
We eliminated the questions regarding computer background. We focused, instead, on the ways 
in which the use of WebCT may have enhanced student motivation, participation, and success in 
the course. 

One of the surprises of this study is that the chat room, discussion forum, and e-mail features 
were not used as extensively by students as they had been in the past. Less than 15% of the stu- 
dents in each class used the communication tools more than face-to-face or telephone communi- 
cation. We would like to develop an environment in which the students and the instructor effec- 
tively use the communication tools to assist in problem clarification, critical thinking, testing and 
debugging, and general problem solving. The students should also use it as a forum for sharing 
ideas and planning study sessions. 

The majority of the students (CSC 121 - 91.4%, CSC 241 - 76.6%, CSC 321 - 90.9%) would 
rather look up grades on WebCT as opposed to getting them from the instructor. The statistics 
are not as high (CSC 121 - 36.2%, CSC 241 - 46.2%, CSC 321 - 40.9%) for students desiring to 
get course information, such as syllabus, notes, and assignments, from WebCT rather than hav- 
ing them distributed in class. However, several said that it depends on what is being distributed 
(CSC 121 - 46.6%, CSC 241 - 38.5%, CSC 321 - 40.9%). 

As in the pre-course survey, the students were asked to indicate which pages, if any, they visited 
during the class. Figure 6 presents the results of the survey and illustrates that almost all of the 
students accessed the assignments (we can only hope that the remaining obtained a copy from a 
friend!). In those courses where lecture notes were available, a majority of the students accessed 
those notes. As indicated earlier, less than the number expected actually utilized the communica- 
tion tools. 

Finally, the students were asked if they will now check courses prior to the first day of class to 
determine if WebCT is being used by the instructor. A surprising number said no (CSC 121 - 
19%, CSC 241 - 15.4%, CSC 321 - 22.7%), while a fairly significant number said it depends on 
the class (CSC 121 - 25.9%, CSC 241 - 30.8%, CSC 321 - 40.9%). 

Opinions on WebCT in the Classroom 

One of the common concerns expressed by educators is that use of a CMS actually demands 
more time (from both students and instructor) than was required before enhancement [2, 8]. With 
this is mind, we have begun to develop a plan that, in addition to using WebCT for the “normal” 
course management functions, focuses on utilizing WebCT as an “information sharing” type of 
resource. 
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Figure 6. Pages Accessed During the Course 
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We have also examined the advantages and disadvantages from both the faculty and the student 
perspective to determine if the use of a course management system is an effective tool for pro- 
moting student learning. Our findings are that, in general, students seem to acclimate fairly 
quickly to the environment. There is a relatively short learning curve, and students seem to enjoy 
using WebCT, as supported by our survey results. 

From the faculty perspective, there are both advantages and disadvantages. The development 
time, especially for a new web-based course, tends to be higher initially. This concern has been 
expressed repeatedly by many researchers [2, 8]. However, as the faculty member becomes more 
accustomed to the system itself, this development time may decrease, even for new courses. We 
have gone from a department in which 2 faculty members (less than 20%) use WebCT to one in 
which 6 (over 50%) are using it to some degree. The advantages of using WebCT may include 
less grading time, less administrative time (especially in those courses taught previously), and 
more personal interaction with students utilizing e-mail and the discussion forum. We have also 
noticed assignment submission rates as high as, and sometimes higher than, those of non-WebCT 
courses [5]. In general, the advantages seem to outweigh the disadvantages for both students and 
faculty. 

Current Assessment and Future Plans 

There are a number of items we would like to address in the future. We would like to continue 
the pre-course/end-of-course survey over a period of time to determine if our findings remain 
consistent. Additionally, we would like to further examine our findings and establish a plan for 
utilizing WebCT in our courses such that we may further enhance the problem-solving and criti- 
cal thinking skills of our students. We will continue to use WebCT in our courses as we believe 
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that, used appropriately, it can in fact be used to enhance (not just supplement) teaching method- 
ology and thus improve student motivation and learning. 
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